Polystore Query Rewriting: The Challenges of Variety

Numerous databases marketed as SQL-on-Hadoop, NewSQL [16] and NoSQL have emerged to catalyze Big Data applications. These databases generally support the 3Vs [7]. (i) Volume: amount of data (ii) Velocity: speed of data in and out (iii) Variety: semi-structured and heterogeneous data. As a result of differing use cases and design considerations around the Variety requirement, these new databases have adopted semi-structured data models that vary among each other. Their query languages have even more variations. Some variations are due to superficial syntactic differences. Some variations arise from the data model differences. Other variations are genuine differences in query capabilities. Yet another kind of variations involves subtly different semantics for seemingly similar query functionalities. E.g., equality may have subtle and unexpected meanings in the presence of missing attributes in NoSQL databases. Even in a single organization, it is common to find multiple databases that exhibit high variety. Often applications require integrated access to those databases. It is difficult to write optimized software that retrieves data from multiple such databases, given the different data models, different query syntaxes and the (often subtly) different query semantics. This problem has been recognized for many decades in the database community. It is now accentuated, as a plethora of different and specialized databases finds its place in the enterprise. For example, the problem happens whenever an enterprise adopts a fast and scalable NoSQL database to capture its users’ activity on its web site (web log) and then builds applications that need integrated access to the web log data stored in the NoSQL database and also to data in its existing SQL databases.